September 2024 update #1004

Jennit07 · 2024-09-17T10:22:25Z

September update released on 13/09/24. We can now merge our September-24-update branch into master.

Specify file paths in sc function parameters

* Update `write_tests_xlsx` * Update documentation * Add in sandpit tests where the extract is saved * Setup tests for sandpit Further checks needed for writing to disk * Update documentation * Amend case_when statement * rename function to include 'sc' * Update documentation * Use `is.null` instead of `missing` * Update documentation * Add `year` as a parameter * Update documentation * Setup for writing sandpit tests to disk * Update parameters for sandpit tests * Update documentation * Use `process_tests_sc_sandpit` * Apply styling * Style code * update documentation Co-authored-by: Zihao Li <[email protected]> * Rename variable sc_id Co-authored-by: Zihao Li <[email protected]> * Rename variable Co-authored-by: Zihao Li <[email protected]> * Rename variable Co-authored-by: Zihao Li <[email protected]> * Update documentation * [check-spelling] Update metadata Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8689503990/attempts/1 Accepted in #943 (comment) Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]> * update spelling * update spelling expect variant --------- Signed-off-by: check-spelling-bot <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Zihao Li <[email protected]> Co-authored-by: Zihao Li <[email protected]>

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8897746003/attempts/1 Accepted in #947 (comment) Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]>

…update?? anyway, now it is fixed.

easy fix but took a long time to identify the issue. Close #944

for some reason the latest scid code was overwritten after the march update?? anyway, now it is fixed.

Collect data before manipulations

adding the death dates to the cases where there is activity after death Co-authored-by: marjom02 <[email protected]>

* adding in syst_time alerts for all functions in create episode and create individual. So that when it runs manually I can see where it is and where it's getting stuck * Style code --------- Co-authored-by: marjom02 <[email protected]> Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: Jennit07 <[email protected]>

Closes #957 and #968

* # major changes to care home script see document on sharepoint for description also: - added in type of admission description - updated care home contact in fill_ch_name script * minor note updates * Style code * Update documentation * couple of note updates * Update R/process_sc_all_care_home.R Co-authored-by: Jennit07 <[email protected]> * Update R/process_sc_all_care_home.R Co-authored-by: Jennit07 <[email protected]> * Style code * Update R/process_sc_all_care_home.R * change to ch name lookup * Update documentation * remove fill ch provider fill line * update fill ch names so it works with new ch methodology * Style code * Update documentation * Style code * Update documentation * Remove redundant variable `latest_sc_id` * use slfhelper::get_chi * new section for sc_ch_id_markers * Style code * Update documentation * Remove extra text and white space * add rename to use death_date_chi * use `read_excel` function * Update documentation * Return the paths only for SPD and ch name * Update documentation * Remove rename - no longer needed * fix typo * remove variables that dont exist --------- Co-authored-by: marjom02 <[email protected]> Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Jennifer Thom <[email protected]>

…urce-linkage-files into september-2024

added in missing variable at the end

In the care home methodology update #945 the cis markers were working but not for episodes nested within a stay, it would wrongly assign the 'next' episode counter. This was due to the lag function assigning NA to the first row. This update fixes the nested cases and i think it is working properly now for all scenarios. Where the first row is NA for `previous_discharge_date` this will use the `ch_discharge_date` which will be the first episode. We also use the following code, adapted from Social care scripts to create the cis markers. This simplifies the code and seems to work for all scenarios. e.g. when the first episode linked to a chi is NA and the next two episodes are continuous, this will return 1,2,2 correctly. Previously this would return 1,2,3. ``` mutate(continuous_marker = accumulate(new_episode[-1], .init = 1, ~ if (.y == "Y") {.x + 1} else {.x})) ```

missing a line from a function.

* Add NSU code to github Includes extracting the service user cohort to send to the chili team and then NSU extraction. * Style code * Add compression and package library * Style code * pick up latest geography file, and save out with compression (#983) * pick up latest geography file, and save out with compression * Style code * use `get_spd_path` Co-authored-by: James McMahon <[email protected]> --------- Co-authored-by: marjom02 <[email protected]> Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: James McMahon <[email protected]> --------- Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Megan McNicol <[email protected]> Co-authored-by: marjom02 <[email protected]> Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: James McMahon <[email protected]>

* removing packages that I don't think get used anywhere. and removing references to fst and spss files * Update documentation * Update authors in description * [check-spelling] Update metadata Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10469297653/attempts/1 Accepted in #984 (comment) Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]> * [check-spelling] Update metadata Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10469690723/attempts/1https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10469690723/attempts/1 Accepted in #984 (comment) Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]> --------- Signed-off-by: check-spelling-bot <[email protected]> Co-authored-by: marjom02 <[email protected]> Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: Jennifer Thom <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Jennit07 <[email protected]>

* update write_tests_xlsx * update process_refined_death * fix tests by removing get_chi * add 2425 * Style code * fix NA matches in refined_death * move latest_cost_year() to cost_uplift() * improve automation * Update documentation * fix `cij_ppa` in DD data * fix bugs of dd and populate cij_delay back to episodes * Style code * keep all variable for delayed discharge episodes * remove dummy variable names from dd_date * Style code * remove `deceased_boxi` variable - bug * remove `create_person_id`. Its matched in client * remove `create_person_id` * Update `run_slf_manually` scripts * further remove person_id * fix duplicate row introduced by adding death * remove duplicated chi when joining death data * TODO: check distinct death data by chi while keeping chi==NA records * add parameter for year * fix duplicate in add_activity_after_death_flag * Update `check_year_valid` * Declare DN variables * Style code * remove redundant variables --------- Co-authored-by: Zihao Li <[email protected]> Co-authored-by: lizihao-anu <[email protected]> Co-authored-by: Jennit07 <[email protected]>

…s. (#988) * fix sc_client_lookup sc_send_lca * fix an issue of get_pop_path * Style code * fix the rest of get_pop_path from get_datazone_pop_path * Update documentation * fix sc_send_lca * add missing year column * Remove redundant code * Update documentation * Style code * explicitly specify the argument year to avoid corruption of targets * Update documentation * Reorder when we match on client variables This was causing NSUs to show a social care id. This now resolves this. * Update documentation * Style code * Add chi parameter to `create_demog_test_flags` * Style code * Use CHI parameter for ep/indiv tests * Use CHI parameter for extract tests (chi) * Change test sheet names to lowercase * Change date to lowercase * Update documentation * new data pipeline with targets remove create_individual_files from targets and append it to run_targets script * minor changes * Style code * Update documentation * Update documentation * Style code * undo sc_send_lca bit * Add code for running years available * Update `_targets.R` script for running old years * Style code * Update `check_year_valid` for running old years * Use `check_year_valid` where no data for old yrs * Style code * Fix pick variables This was not taking the correct variables, leading to NSUs being assigned psychiatry * SC Demographics and SDS (#900) * Style code * # read in sc demographics different variables - removed extract date as not accurate, using chi over upi after discussion with social care data management. Added in date of death just for fun. * social care demographics first draft removed a lot of the submitted variables and instead using chi variables from chi seeding. Other changes: - Fill in missing values, - create flag for latest social care id (one from database is not accurate), this makes sure that each chi only has ONE sc id as the latest to stop it creating duplicates - change postcode to choose chi over submitted * Style code * had a github error? Not sure what happened but commiting first draft of sc demographics * Style code * first draft sds. No major changes - only how demographics is matched on and how latest social care id is selected * Update documentation * demographics - add sending location to group by * Style code * Update documentation * Added ungroup() * Remove comments * Remove comments * Style code --------- Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: marjom02 <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Zihao Li <[email protected]> * Sc all at speedup (#904) * speed up process_sc_all_alarms_telecare function with data.table package * Update documentation --------- Co-authored-by: lizihao-anu <[email protected]> Co-authored-by: Megan McNicol <[email protected]> Co-authored-by: Jennit07 <[email protected]> * Add case_when statement for `high_cc` cohort * Bug - `high_cc` in demographic cohort showing `NAs` instead of `TRUE/FALSE` (#911) Add case_when statement for `high_cc` cohort * added a casewhen to update property type description for homelessness * Update documentation * Style code * Bug - deal with missing variables (#914) * Add missing sc variables for no sc data * Fix code for including `_inc_dna` variables * Remove commented line * Bug - Fix get pop path failing and preventing the indiv file from running. (#913) Fix bug - pop file paths breaking indiv file * correct file hscp file path * Declare missing variables for older years * setup targets scripts for old years * Style code * Include `check_year_valid` for sc client path * Add check year valid to join sc client * Add if else statement * WIP - TO DO - fix dummy path for `get_chi()` * Style code * update dummy data file to read empty tibble * Update `check_year_valid` * Update declared `NA` variables * Update documentation * declare `count_not_known` as NA * supply year as default in `aggregate_by_chi` * Decalre unused variables * Style code * Update sc client with sept update new code * Specify code for running older years * Style code * Add Running SLF files manually scripts * Style code * update write_tests_xlsx * update process_refined_death * fix tests by removing get_chi * add 2425 * Style code * fix NA matches in refined_death * move latest_cost_year() to cost_uplift() * improve automation * Update documentation * fix `cij_ppa` in DD data * fix bugs of dd and populate cij_delay back to episodes * Style code * keep all variable for delayed discharge episodes * remove dummy variable names from dd_date * Style code * remove `deceased_boxi` variable - bug * remove `create_person_id`. Its matched in client * remove `create_person_id` * Update `run_slf_manually` scripts * further remove person_id * fix duplicate row introduced by adding death * remove duplicated chi when joining death data * TODO: check distinct death data by chi while keeping chi==NA records * add parameter for year * fix duplicate in add_activity_after_death_flag * Update `check_year_valid` * Declare DN variables * Style code * Declare client variables * remove extra dd variables * remove redundant variables * remove fy variable * Remove redundant variable `count_not_known` * Remove duplicate code * revert commit - remove fy * update manual run * declare missing sc variables indiv file * Style code --------- Co-authored-by: Zihao Li <[email protected]> Co-authored-by: lizihao-anu <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Megan McNicol <[email protected]> Co-authored-by: SwiftySalmon <[email protected]> Co-authored-by: Zihao Li <[email protected]> Co-authored-by: marjom02 <[email protected]>

github-actions · 2024-09-17T10:58:13Z

@check-spelling-bot Report

🔴 Please review

See the 📂 files view, the 📜action log, or 📝 job summary for details.

Unrecognized words (5)

casewhen
commiting
Decalre
scid
tbc

To accept these unrecognized words as correct, you could run the following commands

... in a clone of the [email protected]:Public-Health-Scotland/source-linkage-files.git repository
on the september-2024 branch (ℹ️ how do I use this?):

curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/main/apply.pl' |
perl - 'https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/10901644059/attempts/1'

OR

To have the bot accept them for you, reply quoting the following line:
@check-spelling-bot apply updates.

Available 📚 dictionaries could cover words (expected and unrecognized) not in the 📘 dictionary

This includes both expected items (307) from .github/actions/spelling/expect.txt and unrecognized words (5)

Dictionary	Entries	Covers	Uniquely
cspell:fullstack/dict/fullstack.txt	419	3	3
cspell:k8s/dict/k8s.txt	153	4	1
cspell:php/dict/php.txt	1689	4
cspell:node/dict/node.txt	891	3	1
cspell:npm/dict/npm.txt	302	3

Consider adding them (in .github/workflows/spelling.yml) for uses: check-spelling/check-spelling@main in its with:

      with:
        extra_dictionaries:
          cspell:fullstack/dict/fullstack.txt
          cspell:k8s/dict/k8s.txt
          cspell:php/dict/php.txt
          cspell:node/dict/node.txt
          cspell:npm/dict/npm.txt

To stop checking additional dictionaries, add (in .github/workflows/spelling.yml) for uses: check-spelling/check-spelling@main in its with:

check_extra_dictionaries: ''

Pattern suggestions ✂️ (1)

You could add these patterns to .github/actions/spelling/patterns.txt:

# Automatically suggested patterns
# hit-count: 3 file-count: 1
# Compiler flags
(?:^|[\t ,"'`=(])-[DPWXYLlf](?=[A-Z]{2,}|[A-Z][a-z]|[a-z]{2,})

Errors (3)

See the 📂 files view, the 📜action log, or 📝 job summary for details.

❌ Errors	Count
ℹ️ candidate-pattern	1
❌ forbidden-pattern	2
❌ ignored-expect-variant	3

See ❌ Event descriptions for more information.

If the flagged items are 🤯 false positives

If items relate to a ...

binary file (or some other file you wouldn't want to check at all).

Please add a file path to the excludes.txt file matching the containing file.

File paths are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your files.

^ refers to the file's path from the root of the repository, so ^README\.md$ would exclude README.md (on whichever branch you're using).
well-formed pattern.

If you can write a pattern that would match it,
try adding it to the patterns.txt file.

Patterns are Perl 5 Regular Expressions - you can test yours before committing to verify it will match your lines.

Note that patterns can't match multiline strings.

Jennit07 and others added 30 commits March 26, 2024 16:10

update documentation

b829eab

Update sc connection name

99f9c2d

Update documentation

298e613

936 - Update parameters with file paths (#939)

fe189a9

Specify file paths in sc function parameters

Merge branch 'master' into June-24-update

6fe98c8

remove and merge overlapping records in GP OoHs

f03c5c6

Style code

fda0c51

update spelling to lowercases

713e7a7

update spelling

a0ac030

only select columns we want in ltc raw data

6baad29

[check-spelling] Update metadata

211c8a7

Update for https://github.com/Public-Health-Scotland/source-linkage-files/actions/runs/8897746003/attempts/1 Accepted in #947 (comment) Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]>

for some reason the latest scid code was overwritten after the march …

86efa00

…update?? anyway, now it is fixed.

only select columns we want in ltc raw data (#947)

08d0066

easy fix but took a long time to identify the issue. Close #944

Style code

d571cb6

latest social care ID update (#948)

fa5cdcd

for some reason the latest scid code was overwritten after the march update?? anyway, now it is fixed.

Merge May24 NI update into June update branch (#949)

a75374e

Collect data before manipulations

Update NEWS.md

08b9f80

link GP-OoH with CUP markers

7f569d3

Style code

20cdf4d

update gp ooh cup

3869b09

link cup to acute

a967d5c

Update documentation

a989b3f

adding the death dates to activity after death cases (#972)

ce90a45

adding the death dates to the cases where there is activity after death Co-authored-by: marjom02 <[email protected]>

Update slf deaths lookup function name

5beee76

automate combined deaths lookup

33278ef

Merge branch 'combined_deaths' into targets_deaths

59235a3

Automate the combined slf deaths lookup (#973)

a7f983a

Closes #957 and #968

Merge branch 'september-2024' into gpooh_cup

1d2562d

lizihao-anu and others added 20 commits August 6, 2024 16:52

Update documentation

03b41c3

update ch_chi_cis methodology

8f4a06a

update ch_sc_id_cis methodology

4c8bbf3

Update notes

b96feb1

Merge branch 'september-2024' of github.com:Public-Health-Scotland/so…

c01b7f6

…urce-linkage-files into september-2024

Use right_join

46a668c

Update process_sc_all_care_home.R

f7caaea

added in missing variable at the end

Latest scid update (#990)

e672d4a

missing a line from a function.

update reference

72a66f8

minor changes to social care code

7146a11

Style code

5e31183

Update process_sc_all_care_home.R

962f7a3

Update NEWS.md

b24399e

update copy_to_hscdiip.R

69ce173

Jennit07 requested a review from lizihao-anu September 17, 2024 10:22

This comment has been minimized.

Sign in to view

Update replace_sc_id_with_latest function

4e3e185

Style code

27ac757

lizihao-anu approved these changes Sep 17, 2024

View reviewed changes

lizihao-anu added this pull request to the merge queue Sep 17, 2024

Merged via the queue into master with commit 4df8dad Sep 17, 2024
1 check passed

lizihao-anu deleted the september-2024 branch September 17, 2024 13:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

September 2024 update #1004

September 2024 update #1004

Jennit07 commented Sep 17, 2024

This comment has been minimized.

github-actions bot commented Sep 17, 2024

September 2024 update #1004

September 2024 update #1004

Conversation

Jennit07 commented Sep 17, 2024

This comment has been minimized.

github-actions bot commented Sep 17, 2024

@check-spelling-bot Report

🔴 Please review

Unrecognized words (5)